Systems and Methods for Synthesizing High Resolution Images Using Image Deconvolution Based on Motion and Depth Information

ABSTRACT

Systems and methods for synthesizing high resolution images using image deconvolution and depth information in accordance embodiments of the invention are disclosed. In one embodiment, an array camera includes a processor and a memory, wherein an image deconvolution application configures the processor to obtain light field image data, determine motion data based on metadata contained in the light field image data, generate a depth-dependent point spread function based on the synthesized high resolution image, the depth map, and the motion data, measure the quality of the synthesized high resolution image based on the generated depth-dependent point spread function, and when the measured quality of the synthesized high resolution image is within a quality threshold, incorporate the synthesized high resolution image into the light field image data.

CROSS-REFERNCE TO RELATED APPLICATIONS

The current application is a continuation of U.S. patent applicationSer. No. 14/200,629, entitled “Systems and Methods for Synthesizing HighResolution Images Using Image Deconvolution Based on Motion and DepthInformation” to Lelescu et al., filed Mar. 7, 2014, which applicationclaims priority to U.S. Provisional Patent Application Ser. No.61/788,078, filed Mar. 15, 2013, the disclosure of which is herebyincorporated by reference in its entirety.

FIELD OF THE INVENTION

The present invention relates to systems and methods for imagedeconvolution and more specifically to synthesizing high resolutionimages using image deconvolution processes.

BACKGROUND

Imaging devices, such as cameras, can be used to capture images ofportions of the electromagnetic spectrum, such as the visible lightspectrum, incident upon an image sensor. For ease of discussion, theterm light is generically used to cover radiation across the entireelectromagnetic spectrum. In a typical imaging device, light entersthrough an opening (aperture) at one end of the imaging device and isdirected to an image sensor by one or more optical elements such aslenses. The image sensor includes pixels or sensor elements thatgenerate signals upon receiving light via the optical element. Commonlyused image sensors include charge-coupled device (CCDs) sensors andcomplementary metal-oxide semiconductor (CMOS) sensors. Each pixel in animage sensor is capable of capturing light and converting the capturedlight into electrical signals. In order to separate the colors of lightand capture a color image, a Bayer filter is often placed over the imagesensor, filtering the incoming light into its red, blue, and green (RGB)components that are then captured by the image sensor. The RGB signalcaptured by the image sensor using a Bayer filter can then be processedand a color image can be created.

SUMMARY OF THE INVENTION

Systems and methods for synthesizing high resolution images using imagedeconvolution and depth information in accordance embodiments of theinvention are disclosed. In one embodiment, an array camera includes aprocessor and a memory connected to the processor and configured tostore an image deconvolution application, wherein the imagedeconvolution application configures the processor to obtain light fieldimage data, where the light field image data includes a synthesized highresolution image having a plurality of pixels, a depth map, and metadatadescribing the motion associated with a capturing device that capturedthe light field image data, determine motion data based on the metadatacontained in the light field image data, generate a depth-dependentpoint spread function based on the synthesized high resolution image,the depth map, and the motion data, where the depth-dependent pointspread function describes the blurriness of points within thesynthesized high resolution image based on the motion of the capturingdevice and the depth of the pixels described in the depth map, measurethe quality of the synthesized high resolution image based on thegenerated depth-dependent point spread function, when the measuredquality of the synthesized high resolution image is within a qualitythreshold, incorporate the synthesized high resolution image into thelight field image data, and when the measured quality of the synthesizedhigh resolution image is outside the quality threshold, determineupdated motion data based on the measured quality and thedepth-dependent point spread function, generate an updateddepth-dependent point spread function based on the updated motion data,and synthesize a new high resolution image based on the updateddepth-dependent point spread function.

In an additional embodiment of the invention, the array camera furtherincludes an array camera module including an imager array havingmultiple focal planes and an optics array configured to form imagesthrough separate apertures on each of the focal planes, wherein thearray camera module is configured to communicate with the processor, andwherein the obtained light field image data includes image data capturedby the imager array.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a system diagram of an array camera including a 5×5 imagerarray with storage hardware connected with a processor in accordancewith an embodiment of the invention.

FIG. 2 is a functional block diagram of an image deconvolution deviceconfigured to perform image deconvolution processes on synthesizedimages in accordance with an embodiment of the invention.

FIG. 3 is a flow chart illustrating a process for blind imagedeconvolution of a synthesized image using depth information inaccordance with an embodiment of the invention.

FIG. 4 is a flow chart illustrating a process for image deconvolution ofa synthesized image using estimated camera motion in accordance with anembodiment of the invention.

FIG. 5 is a flow chart illustrating a process for image deconvolution ofa synthesized image using measured camera motion in accordance with anembodiment of the invention.

FIG. 6 is a flow chart illustrating a process for determining an apriori rigid motion transformation of an imager array in accordance withan embodiment of the invention.

FIG. 7 is a flow chart illustrating a process for generating adepth-dependent point spread function in accordance with an embodimentof the invention.

DETAILED DESCRIPTION

Turning now to the drawings, systems and methods for performing imagedeconvolution processes utilizing depth information in accordance withembodiments of the invention are illustrated. Array cameras, such asthose described in U.S. patent application Ser. No. 12/935,504, entitled“Capturing and Processing of Images using Monolithic Camera Array withHeterogeneous Imagers” to Venkataraman et al., can be utilized tocapture light fields and store the captured light fields. In a varietyof embodiments, a light field is captured by obtaining image dataforming a two dimensional array of image data of a scene captured frommultiple viewpoints. In accordance with many embodiments of theinvention, images, including high resolution images, may be synthesizedfrom image data captured by an appropriately configured array camerausing a super-resolution process. Systems and methods for performingsuper-resolution processing on image data captured by an array camera inaccordance with embodiments of the invention are disclosed in U.S.patent application Ser. No. 12/967,807 entitled “System and Methods forSynthesizing High Resolution Images Using Super-Resolution Processes” toLelescu et al. In many instances, the process of synthesizing a highresolution image may result in a single image, a stereoscopic pair ofimages that can be used to display three dimensional (3D) informationvia an appropriate 3D display, and/or a variety of images from differentviewpoints. The process of synthesizing high resolution images fromlower resolution image data captured by an imager array in an arraycamera typically involves performing parallax detection and correctionto reduce disparity between the images captured by each of the camerasin the array camera module. Processes for parallax detection andcorrection in accordance with embodiments of the invention are disclosedin U.S. patent application Ser. No. 13/972,881 entitled “Systems andMethods for Parallax Detection and Correction in Images Captured UsingArray Cameras that Contain Occlusions using Subsets of Images to PerformDepth Estimation” to Venkataraman et al. In many embodiments, processesfor performing parallax detection and correction yield depthinformation. Therefore, processes for synthesizing high resolutionimages using lower resolution image data using super-resolutionprocesses can also involve the generation of a depth map describing thedepth information. The term depth map refers to metadata that describesthe distance from the camera to the portion of the scene captured in thepixels (or a subset of the pixels) of an image.

Image data captured using a camera are often blurry due to the qualityof the sensor, movement of the camera, hand jitter, and/or other factorsaffecting the quality of the captured image data. The flaws in blurryimages can be further exacerbated when the blurry images are used tosynthesize images using super-resolution processes, resulting in imagescontaining flaws influencing the blurriness and quality of thesynthesized image. In particular, the flaws in the captured image dataaffect the quality of the synthesized image depend based in part on thedepth of the flaws within the synthesized image. Note that, in a varietyof embodiments, the synthesized image need not be synthesized usingsuper-resolution processes; rather any image synthesized from image datacontaining depth information can be affected by flaws in the capturedlight field due to the motion of the capturing device during thecapture.

Image deconvolution devices (such as array cameras configured to viasoftware to perform image deconvolution processes) in accordance withembodiments of the invention are configured to perform imagedeconvolution processes on synthesized images to improve the quality ofthe synthesized image by compensating for the motion of the deviceduring the capturing of a light field from multiple viewpoints fromwhich the image is synthesized. Image deconvolution devices areconfigured to perform image deconvolution processes by generating adepth-dependent (and/or motion dependent) point spread function (PSF)based on the motion of the array camera capturing the image datautilized in the synthesis of the image being deconvoluted and the depthinformation determined in the performance of parallax detection andcorrection in synthesizing the image. A depth-dependent PSF describesthe sharpness and/or blurriness of points within a synthesized imagebased on the motion of the capturing device and the depth of the points.PSFs utilize blur kernels to describe the blur for the points within thesynthesized image. A blur kernel is a function describing the sharpness(and/or blur) of the area surrounding a point within a synthesizedimage. The motion of the array camera can be determined using a blindmotion estimation process, an ego-motion estimation of the array cameramotion, and/or the motion of the array camera as measured by the arraycamera. Although the processes described herein are in reference toperforming image deconvolution processes on entire images, it should benoted that the processes described herein can be performed on tiles orpatches of images in accordance with embodiments of the invention.

The disclosures of each of U.S. patent application Ser. Nos. 12/935,504,12/967,807, and 13/972,881 are hereby incorporated by reference in theirentirety. Although the systems and methods described are with respect toarray cameras configured to both capture and process captured lightfields, devices (including image deconvolution devices that are notconfigured to capture light fields) that are configured to obtaincaptured light fields captured using a different device and process thereceived data can be utilized in accordance with the requirements of avariety of embodiments of the invention. Additionally, any of thevarious systems and processes described herein can be performed insequence, in alternative sequences, and/or in parallel (e.g. ondifferent computing devices) in order to achieve similar results in amanner that is more appropriate to the requirements of a specificapplication of the invention. Systems and methods for performing imagedeconvolution processes on synthesized images in accordance withembodiments of the invention are described below.

Array Camera Architectures

Array cameras in accordance with many embodiments of the inventioninclude sensors such as, but not limited to, an inertial movement sensorsystem or inertial measurement unit, configured to measure the motion ofthe array camera at the time a light field is captured. An array cameraincluding an imager array in accordance with an embodiment of theinvention is illustrated in FIG. 1. The array camera 100 includes anarray camera module including an imager array 102 having multiple focalplanes 104 and an optics array configured to form images throughseparate apertures on each of the focal planes. The imager array 102 isconfigured to communicate with a processor 108. In accordance with manyembodiments of the invention, the processor 108 is configured to readout image data captured by the imager array 102 and synthesize imagesusing super-resolution processes. Imager arrays including multiple focalplanes are discussed in U.S. patent application Ser. No. 13/106,797,entitled “Architectures for System on Chip Array Cameras” to McMahon etal., the entirety of which is hereby incorporated by reference.

In the illustrated embodiment, the focal planes are configured in a 5×5array. In other embodiments, any of a variety of array configurationscan be utilized including linear arrays and subsets of the array. Eachfocal plane 104 of the imager array is capable of capturing image datafrom an image of the scene formed through a distinct aperture.Typically, each focal plane includes a plurality of rows of pixels thatalso forms a plurality of columns of pixels, and each focal plane iscontained within a region of the imager that does not contain pixelsfrom another focal plane. The pixels or sensor elements utilized in thefocal planes can be individual light sensing elements such as, but notlimited to, traditional CIS (CMOS Image Sensor) pixels, CCD(charge-coupled device) pixels, high dynamic range sensor elements,multispectral sensor elements and/or any other structure configured togenerate an electrical signal indicative of light (e.g. anyelectromagnetic signal) incident on the structure. In many embodiments,the sensor elements of each focal plane have similar physical propertiesand receive light via the same optical channel and color filter (wherepresent). In other embodiments, the sensor elements have differentcharacteristics and, in many instances, the characteristics of thesensor elements are related to the color filter applied to each sensorelement. In a variety of embodiments, a Bayer filter pattern of lightfilters can be applied to one or more of the focal planes 104.Similarly, the sensor elements can be configured to capture informationfrom non-visible light sources, such as infrared as appropriate to thespecific requirements of embodiments of the invention.

In a variety of embodiments, the processor 108 is connected to aninertial movement sensor system 110 that is configured to capture motioninformation related to the rotation and/or translation of the arraycamera. A variety of inertial measurement sensors can be utilized inaccordance with a number of embodiments of the invention including, butnot limited to, gyroscopes and microelectromechanical system (MEMS)accelerometers. In several embodiments, the inertial movement sensorsystem 110 is configured to measure the rotation and/or translation ofthe imager array 102. In many embodiments, measuring the rotation and/ortranslation of the imager array 102 involves measuring the rotationand/or translation of the focal planes 104. In a number of embodiments,the inertial movement sensor 110 is integrated into or connected to theimager array 102 and the processor 108 is configured to receive therotation and/or translation information from the imager array 102. Inorder to calibrate the inertial movement sensor system 110 to the imagerarray 102 (and/or focal planes 104), an a priori rigid motiontransformation is determined that calibrates the motion data determinedusing the inertial movement sensor 110 to the movement of the imagerarray 102. Processes for determining an a priori rigid motiontransformation in accordance with embodiments of the invention arediscussed below.

In several embodiments, information captured by one or more focal planes104 is read out of the imager array 102 as packets of image data. Inseveral embodiments, a packet of image data contains one or more pixelsfrom a row of pixels captured from each of one or more of the focalplanes 104. Packets of image data may contain other groupings ofcaptured pixels, such as one or more pixels captured from a column ofpixels in each of one or more focal planes and/or a random sampling ofpixels. Systems and methods for reading out image data from arraycameras that can be utilized in array cameras configured in accordancewith embodiments of the invention are described in U.S. Pat. No.8,305,456 entitled “Systems and Methods for Transmitting and ReceivingArray Camera Image Data” to McMahon, the disclosure of which isincorporated by reference herein in its entirety.

In a number of embodiments, the captured packets of image data areassociated with an image packet timestamp. In many embodiments, inertialmovement data captured using the inertial movement sensor system 110 isassociated with an inertial movement measurement timestamp. In severalembodiments, the processor 108 associates one or more packets of imagedata with inertial movement measurement data based upon the image packettimestamp(s) and the inertial movement measurement timestamp. In severalembodiments, there is a delay in the capture between the first pixelcaptured by the focal plane 104 and the last pixel captured. As isdiscussed further below, image deconvolution processes in accordancewith embodiments of the invention utilize the timestamps associated withthe image data and the inertial measurement data to accommodatevariations in motion during the period in time in which image datautilized in synthesizing a high resolution image is captured.

Array cameras are configured to create a light field image data filecontaining a synthesized high resolution image and metadata generatedusing the captured light field image data. One such file format is theJPEG-DX extension to ISO/IEC 10918-1 described in U.S. patentapplication Ser. No. 13/631,731, titled “Systems and Methods forEncoding Light Field Image Files” to Venkataraman et al, the entirety ofwhich is hereby incorporated by reference. The metadata can include avariety of data captured and/or generated by the array camera,including, but not limited to, the generated depth information, thetimestamp information, and/or motion data describing the motion of thearray camera during the capture of the image data.

Although a specific array camera utilized in array camera configured toperform image deconvolution processes is illustrated in FIG. 1,alternative architectures can also be utilized as appropriate to therequirements of specific applications in accordance with embodiments ofthe invention. Systems and methods for synthesizing high resolutionimages using image deconvolution processes in accordance withembodiments of the invention are discussed below.

Image Deconvolution Devices

A variety of devices can be configured to synthesize high resolutionimages using captured light fields and image deconvolution processes,including array cameras and other devices that are not capable ofcapturing image data using an imager array. A diagram of an imagedeconvolution device in accordance with an embodiment of the inventionis illustrated in FIG. 2. The image deconvolution device 200 contains aprocessor 210 capable of being configured via software (e.g. an imagedeconvolution application) to load and manipulate light field image datafiles. In many embodiments of the invention, the processor 210 isconnected to an imager array 220 capable of capturing light fields. In avariety of embodiments, the imager array 220 is configured to capturelight field image data utilizing processes similar to those describedabove.

In many embodiments, the processor 210 is connected to a display 212capable of displaying two dimensional images synthesized using acaptured light field. In several embodiments, the display 212 is capableof displaying three dimensional images. In a number of embodiments, theprocessor 210 is connected to a user interface 214, such as atouchscreen interface, a pointing device, and/or a keyboard. In manyembodiments, the user interface 214 can be a camera or array cameracapable of tracking user movements. In several embodiments, theprocessor 210 is connected to a storage device 216. The storage device216 is capable of storing light field image data files and synthesizedimages and delivering the stored data to the processor 210 formanipulation. In a number of embodiments, the processor 210 is connectedto a network interface 218 capable of communication via a network. Thenetwork communication involves receiving and transmitting light fieldimage data files and/or synthesized images, where the light field imagedata files and/or synthesized images can be stored in storage device 216(if present) or can be loaded directly into the processor 210.

In several embodiments, the processor 210 is configured to synthesizeimages using image data captured using the imager array utilizingsuper-resolution processes as described above. The processor 210 canperform image deconvolution processes on the synthesized images (or anyother images including depth information) using a depth-dependent pointspread function (PSF) determined using motion and depth information. Theprocessor 210 can determine the motion information using a variety ofprocesses, including, but not limited to, blind motion estimation,ego-motion estimation, or using measured motion information contained inmetadata in the light field image data files. Processes for determiningdepth-dependent point spread functions and measuring the motion of anarray camera are described in detail below. In a number of embodiments,input received from the user interface 214 can be utilized in the imagedeconvolution process.

Although a specific example of an array camera configured to performimage deconvolution processes is described above with respect to FIG. 2,any device capable performing image deconvolution processes on imageshaving depth information can be used in accordance with embodiments ofthe invention. Processes for image deconvolution of synthesized imagesin accordance with embodiments of the invention are discussed below.

Blind Image Deconvolution

In order to create a sharp image, image deconvolution devices areconfigured to perform image deconvolution processes on images havingdepth information. However, many images do not have informationregarding the motion of the capturing device. Image deconvolutiondevices in accordance with embodiments of the invention are configuredto perform a blind estimation of the motion of the capturing deviceutilized in the image deconvolution process. A process for performingblind image deconvolution in accordance with an embodiment of theinvention is illustrated in FIG. 3. The process 300 includes obtaining(310) light field image data. An image is synthesized (312). Adepth-dependent point spread function is generated (314) and imagedeconvolution is performed (316) on the synthesized image. If thequality of the synthesized image is within (318) a threshold value, theprocess completes. If the quality of the synthesized image is not within(318) the threshold, the motion estimation is adjusted (320) and a newdepth-dependent point spread function is generated (314).

In a variety of embodiments, the light field image data is obtained(310) directly from an array camera utilizing processes similar to thosedescribed above. In a number of embodiments, the light field image datais obtained (310) from a light field image data file. In severalembodiments, the synthesized (312) image is contained in the light fieldimage data. In many embodiments, an image is synthesized (312) using thelight field image data. The synthesized (312) image contains metadata,where the metadata includes a depth map describing depth informationrelated to the points within the synthesized image. A depth-dependentPSF is generated (314) using the depth map and a motion estimation.Processes for generating depth-dependent PSF in accordance withembodiments of the invention are discussed below. The camera motionestimation can be any predefined value (including no motion at all) asappropriate to the requirements of a specific application in accordancewith embodiments of the invention. Image deconvolution is performed(316) using the depth-dependent PSF and the quality of the resultingimage is analyzed. If the quality of the image is within (318) a qualitythreshold, the image is considered of sufficient quality and the process300 completes. The quality threshold can be dependent on any visualproperties of the image as appropriate to the requirements of a specificapplication in accordance with embodiments of the invention, includingthe sharpness of the image. The quality threshold can be pre-determinedand/or determined dynamically. If the image quality is not within thethreshold (318), the motion estimation is adjusted (320). A number ofprocesses can be utilized to adjust (320) the motion estimation, such asgradient ascent, gradient descent, or any other iterative optimizationtechnique as appropriate to the requirements of a specific applicationin accordance with embodiments of the invention.

Although a specific process for performing blind image deconvolution ofsynthesized images having depth information in accordance with anembodiment of the invention is described above with respect to FIG. 3,processes appropriate to the requirements of specific applications canbe utilized in accordance with embodiments of the invention. Processesfor performing image deconvolution using estimated and/or measuredcamera motion in accordance with embodiments of the invention arediscussed below.

Image Deconvolution with Estimated Motion Information

A variety of image deconvolution devices are configured to estimate themotion of the camera capturing the image being deconvoluted. A processfor performing image deconvolution using estimated motion information inaccordance with an embodiment of the invention is illustrated in FIG. 4.The process 400 includes obtaining (410) light field image data. Imagesfrom multiple viewpoints are synthesized (412). Camera motion isestimated (414), a depth-dependent point spread function is generated(416), and image deconvolution is performed (418).

In several embodiments, obtaining (410) light field image data usingprocesses similar to those described above. Synthesizing (412) imagesfrom multiple viewpoints includes performing a number ofsuper-resolution processes using the light field image data and theassociated depth map to create multiple views of the same scene.Estimating (414) camera motion is performed using ego-motion techniquesin accordance with embodiments of the invention. Ego-motion techniquesestimate camera motion by detecting features in a first image andmatching those features to a second image. This matching process isperformed using pairs of the synthesized (412) images. Based on thematching of the features, an optical flow field is generated; theoptical flow field describes the estimate of the motion of the camera.Other techniques for estimating (414) camera motion can be utilized asappropriate to the requirements of a specific application in accordancewith embodiments of the invention. The depth-dependent point spreadfunction is generated (416) using the estimated camera motion and thedepth map. Image deconvolution is performed (418) using processessimilar to those described above.

A specific process for performing image deconvolution using estimatedcamera motion in accordance with an embodiment of the invention isdescribed above with respect to FIG. 4; however, a variety of processescan be utilized in accordance with embodiments of the invention.Processes for performing image deblurring using depth information,determining the motion of array cameras, and generating point spreadfunctions are described below.

Image Deconvolution with Measured Motion Information

Array cameras in accordance with embodiments of the invention areconfigured to capture motion data describing the motion of the arraycamera at the time image data is captured. A process for performingimage deconvolution using measured motion information in accordance withan embodiment of the invention is illustrated in FIG. 5. The process 500includes obtaining (510) light field image data and synthesizing (512)an image. The measured camera motion is determined (514), adepth-dependent point spread function is determined (516), and imagedeconvolution is performed (518).

In a number of embodiments, obtaining (510) light field image data andsynthesizing (512) an image are performed utilizing processes similar tothose described above. The measured camera motion is determined (514)using motion information contained in the light field image data. Inmany embodiments, the motion information is directly stored as metadatain the light field image data. In a variety of embodiments, the motioninformation is determined (514) by correlating motion information alongwith time information contained in the light field image data. Thecorrelation of the motion information with the time informationdescribes how quickly the array camera moved as packets of image dataincluded in the light field image data were read from the focal planesin the imager array of the array camera. The determined (514) measuredmotion information and the depth map are utilized to generate (516) thedepth-dependent point spread function. In several embodiments, imagedeconvolution is performed (518) utilizing processes similar to thosedescribed above.

A specific process for performing image deconvolution of light fieldimages using depth and camera motion information in accordance with anembodiment of the invention is described above with respect to FIG. 5;however, a variety of processes can be utilized in accordance withembodiments of the invention. Processes for determining the motion ofarray cameras and generating point spread functions are described below.

Calibrating the Motion of an Array Camera

As discussed above, the motion of an array camera capturing image datais utilized in the deconvolution of images captured using the arraycamera. Many array cameras include motion sensors configured todetermine the motion of the array camera as image data is captured. Inorder to align the motion sensor to the captured image data, an a prioririgid motion transformation is determined to calibrate the motion datacaptured by the sensor with the actual movement of the array camera.This a priori rigid motion transformation can be stored in a variety ofmanners, including as metadata associated with a captured light field,for use in later image processing. A process for determining an a prioririgid motion transformation for use in an array camera configured toperform image deconvolution processes in accordance with an embodimentof the invention is illustrated in FIG. 6. The process 600 includescalibrating (610) the location of an imager array contained in an arraycamera. The array camera is moved (612). The path of the array camera iscorrelated (614) with motion information obtained from an inertialmeasurement sensor. The a priori rigid motion transformation isdetermined (616).

In a number of embodiments, calibrating (610) the array camera includesperforming intrinsic and extrinsic camera calibration. Systems andmethods for calibrating a camera module in an array camera are disclosedin U.S. Provisional Patent Application Ser. No. 61/780,748, titled“Systems and Methods for Calibration of an Array Camera” and filed Mar.13, 2013, the entirety of which is hereby incorporated by reference. Ina variety of embodiments, the movement (612) of the camera can be randomand/or a pre-determined path. In accordance with many embodiments of theinvention, captured light fields contain a two dimensional array of twodimensional images. These two dimensional images contain a plurality ofpixels, where each pixel is associated with a coordinate representingwhere that pixel exists in the captured light field and/or a timestamprepresenting the time that the pixel was captured. In many embodiments,the array camera captures one or more light fields while the arraycamera is moved (612). In a variety of embodiments, the path of thearray camera as it is moved (612) is determined by mapping the motion ofone or more pixels between the captured light fields. In severalembodiments, the path of the array camera as it is moved (612) isdetermined using one or more pixels in a captured light field.

In a variety of embodiments, correlating (614) the motion information tothe pixel motion utilizes motion timestamps associated with the capturedmotion information and pixel timestamps associated with the capturedpixels. In a number of embodiments, determining (616) the a priori rigidmotion transformation is a function that correlates (614) the motioninformation to the mapped pixel motion within a predefined errorthreshold. In many embodiments, the error threshold is dynamicallydetermined. The a priori rigid motion transformation includes, but isnot limited to, rotation information and translation information.

In several embodiments, inertial measurement sensors include, but arenot limited to, gyroscopes, accelerometers, and magnetometers. In anumber of embodiments, an array camera contains a plurality of inertialmeasurement sensors. In many embodiments, inertial measurement sensorshave measurement errors and/or accumulate errors over time due to sensornoise. In accordance with embodiments of the invention, the path of thearray camera can be correlated (614) using one or more pixels in eachcaptured light field. By correlating (614) the motion data with thepixels in successive captured light fields, errors due to inertialmeasurement sensor error can be held within a predefined margin oferror. In many embodiments, correlating (614) the motion data isperformed using pixels in a single captured light field. In severalembodiments, the margin of error is dynamically determined. In severalembodiments, the process 600 can be performed once, for example, at thetime the array camera is manufactured. In a variety of embodiments,process 600 can be performed during the use of the array camera torefine the calibration of the motion sensor by updating the a prioririgid motion transformation based on changes in the array camera.

A specific process for determining an a priori rigid motiontransformation for use in an array camera configured to perform imagedeconvolution processes in accordance with an embodiment of theinvention is described above with respect to FIG. 6; however, a varietyof processes can be utilized to determine a priori rigid motiontransformations as appropriate to the requirements of a specificapplication in accordance with embodiments of the invention. Processesfor generating point spread functions in accordance with embodiments ofthe invention are discussed below.

Generating a Point Spread Function

A PSF describes the response of an imaging system to a point object. Forexample, an imaging system may not be able to precisely capture a pointobject, resulting in the blurring of the point object across more thanone pixel on a focal plane of an imager array. In images with depthinformation, the blurring of point objects depends on the depth of thepoint object from the focal plane of the image. Image deconvolutiondevices configured to perform image deconvolution processes inaccordance with embodiments of the invention utilize depth-dependentPSFs to correct for the blurring of point objects in synthesized imagesbased on the depth of the point objects within the synthesized image. Aprocess for generating a point spread function using depth informationin accordance with an embodiment of the invention is illustrated in FIG.7. The process 700 includes computing (710) a depth map for a capturedlight field. Camera motion is determined (712). The pixel path iscalculated (714) for one or more pixels in the captured light field. Ablur kernel is generated (716) for the one or more pixels.

Systems and methods for computing (710) depth maps that can be utilizedin accordance with a variety of embodiments of the invention aredescribed in U.S. Patent Application Serial No. 13/972,881, incorporatedby reference above. Other systems and methods for computing (710) depthmaps can be utilized in accordance with the requirements of a number ofembodiments of the invention. In many embodiments, camera motion isdetermined (712) using an inertial measurement sensor. In severalembodiments, processes similar to those described above can be utilizedto determine (712) camera motion. In a number of embodiments, cameramotion is determined using ego-motion estimation algorithms. In manyembodiments, generating (714) the pixel path involves determining themovement of one or more pixels in a scene using the camera motion. In anumber of embodiments, the camera motion is an a priori rigid motiontransformation generated using a process such as that described above.In several embodiments, the blur kernel is generated (716) using thepixel path calculated (714) for the one or more pixels. In manyembodiments, the calculated (714) pixel path and the blur kernelgenerated (716) are for pixels at varying depth levels in the capturedlight field. In a variety of embodiments, the blur kernel is generated(716) utilizing pixel timestamps associated with one or more pixelsand/or motion timestamps associated with motion data captured using theinertial measurement sensor.

A point p in a scene is located at a location p(x,y,z) , where (x,y,z)represents the three-dimensional location of the point in the scene.Applying an a priori rigid motion transformation having rotation R andtranslation T to a point p in a light field will result in the point pmoving to

M _(R,T)(p)=(RT)p

and the imaged position q=P(x,y,z) will move to

PM_(R,T)(p)

where P is the projective transformation of the array camera capturingthe light field. In several embodiments, the projective transformationof the array camera includes scene independent geometric corrections andscene dependent geometric transformations of the array camera asdescribed in U.S. Provisional Patent Application Ser. No. 61/780,748,the entirety of which is incorporated by reference above.

An a priori rigid motion transformation M_(R,T) is represented by arotation R_(n) and a translation T_(n) for a time sampling period N=1, .. . , n. Appling the a priori rigid motion transformation to pixel pover the time sampling period calculates (714) pixel path

q _(n) =PM _(n)(p)

where

M _(n)(p)=P(R _(n) T _(n) . . . R ₁ T ₁)

Point spread functions contain a blur kernel that describes the responseof the array camera to a point object. In accordance with embodiments ofthe invention, the blur kernel K for a pixel path q_(n) for a timesampling period N=1, . . . , n is generated (716) using the formula

$K = {c{\sum\limits_{n = 1}^{N}\; {G\left( {{q_{n} - q_{1}},\sigma} \right)}}}$

where c is a nominalization constant and G(q_(n)−q₁, σ) is a twodimensional radial symmetric Gaussian distribution with mean q_(n)−q₁and standard deviation σ. In a number of embodiments, G(q_(n)−q₁, σ) isa variety of probability distributions known to those of ordinary skillin the art. In several embodiments, applying the Gaussian distributionto the pixel path generates (716) a depth-dependent spatially varyingblur kernel utilized in a PSF for each pixel depending on the depth ofthe pixel in the light field. In many embodiments, the depth of eachpixel is determined using parallax information determined usingprocesses as described in U.S. patent application Ser. No. 13/972,881,the entirety of which is incorporated by reference above.

A specific process for generating a point spread function using depthinformation in accordance with an embodiment of the invention isdescribed above with respect to FIG. 7; however, a variety of processescan be utilized in accordance with embodiments of the invention.

Although the present invention has been described in certain specificaspects, many additional modifications and variations would be apparentto those skilled in the art. It is therefore to be understood that thepresent invention can be practiced otherwise than specifically describedwithout departing from the scope and spirit of the present invention.Thus, embodiments of the present invention should be considered in allrespects as illustrative and not restrictive. Accordingly, the scope ofthe invention should be determined not by the embodiments illustrated,but by the appended claims and their equivalents.

What is claimed is:
 1. An array camera, comprising: a processor; and amemory connected to the processor and configured to store an imagedeconvolution application; wherein the image deconvolution applicationconfigures the processor to: obtain light field image data, where thelight field image data comprises a synthesized high resolution imagehaving a plurality of pixels, a depth map, and metadata describing themotion associated with a capturing device that captured the light fieldimage data; determine motion data based on the metadata contained in thelight field image data; generate a depth-dependent point spread functionbased on the synthesized high resolution image, the depth map, and themotion data, where the depth-dependent point spread function describesthe blurriness of points within the synthesized high resolution imagebased on the motion of the capturing device and the depth of the pixelsdescribed in the depth map; measure the quality of the synthesized highresolution image based on the generated depth-dependent point spreadfunction; when the measured quality of the synthesized high resolutionimage is within a quality threshold, incorporate the synthesized highresolution image into the light field image data; and when the measuredquality of the synthesized high resolution image is outside the qualitythreshold; determine updated motion data based on the measured qualityand the depth-dependent point spread function; generate an updateddepth-dependent point spread function based on the updated motion data;and synthesize a new high resolution image based on the updateddepth-dependent point spread function.